MapReduce Functions on GasDay Data Using Hadoop

The GasDay lab at Marquette University forecasts natural gas consumption for 26 Local Distributing Companies around the United States. We have a very large amount of data that has accumulated over the past 19 years, and the lab needs a way to select and process from all of this data to gain insight into our forecasting methods. MapReduce is a pair of functions originally proposed by Jeffrey Dean and Sanjay Ghemawat of Google that allows users to process very large data sets. Hadoop is a software framework created by Doug Cutting in 2004 that allows scientists to distribute vast amounts of data across a cluster of many Linux machines and run MapReduce over the entire data set quickly. The quick turn-around facilitates processing of “Big Data.” As examples of MapReduce processing, we found the most accurate type of model, most accurate forecast number, forecast accuracy over time, and the total amount of gas forecasted each year. This is important because we are able to explore different High Performance Computing solutions for anyone with “Big Data.” This allows GasDay to improve our forecasting models and grow continuously as a business. ∗This work was supported by the National Science Foundation under grant CNS-1063041.